feat: add support for alert auto-resolve + Incident.io integration #1298

wrn14897 · 2025-10-25T23:44:53Z

Plus fixed 'group-by' alert state issues (alert histories)

Ref: HDX-2661
Ref: HDX-2660

changeset-bot · 2025-10-25T23:44:56Z

🦋 Changeset detected

Latest commit: 3f3231c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/common-utils	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2025-10-25T23:44:58Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
hyperdx-v2-oss-app	Ready	Preview	Comment	Nov 4, 2025 10:53pm

claude · 2025-10-25T23:46:37Z

Code Review

Critical Issues

❌ Missing index for group-by queries → Add compound index { alert: 1, group: 1, createdAt: -1 } to AlertHistory model

The getPreviousAlertHistories aggregation groups by both alert AND group (line 627-631 in index.ts)
Current index only covers { alert: 1, createdAt: -1 } (alertHistory.ts:58)
For group-by alerts with many groups, MongoDB will need to scan all alert histories and do an in-memory group, degrading performance

Important Issues

⚠️ Potential race condition in skip logic → Consider adding alert-level locking or idempotency keys

Lines 209-222 check if any history exists in current window to skip processing
If multiple workers process the same alert concurrently, both could pass the skip check before either creates history
This could result in duplicate notifications being sent

⚠️ AlertInput.id field lacks validation → Add Zod schema validation or document usage

Added optional id?: string field to AlertInput type (controllers/alerts.ts:21)
No validation ensuring it is a valid ObjectId format
Usage in template.ts:267 assumes it exists without null check

Minor Observations

✅ Test coverage is extensive (1845 lines added to checkAlerts.test.ts)
✅ Auto-resolve logic for missing groups is well-implemented
✅ Good separation of concerns with composite keys for grouped alerts
✅ IncidentIO integration follows existing webhook pattern

Recommendation

Fix the missing compound index before merge - this is a performance issue that will impact production workloads with group-by alerts.

github-actions · 2025-10-25T23:50:56Z

E2E Test Results

✅ All tests passed • 39 passed • 3 skipped • 304s

Status	Count
✅ Passed	39
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	3

View full report →

wrn14897 · 2025-10-28T08:32:22Z

packages/app/src/TeamPage.tsx

+  "title": "{{title}}",
+  "description": "{{body}}",
+  "deduplication_key": "{{eventId}}",
+  "status": "{{#if (eq state "ALERT")}}firing{{else}}resolved{{/if}}",


need to remap the status (https://api-docs.incident.io/tag/Alert-Events-V2#operation/Alert%20Events%20V2_CreateHTTP!path=status&t=request)

pulpdrew

Besides the other comments, I also notice that since we create an alert history for each group, the Alerts page has an alert status per group, which is probably not correct:

For example, if there are three group keys, you'll get three red bars here for the same time and with different counts.

Just a thought - would it make more sense to save one alert history with all the groups, and maintain the state of each group within the one history entry? Sort of like how we have lastValues already? That could cut down on requests to mongo, help avoid making changes to the UI to fix the above ^^ issue, and maybe simplify some of the other code (particularly the stuff that handles partial mongo save failures)?

packages/api/src/tasks/checkAlerts/template.ts

packages/api/src/tasks/checkAlerts/providers/index.ts

packages/api/src/tasks/checkAlerts/index.ts

packages/api/src/tasks/checkAlerts/providers/default.ts

pulpdrew

LGTM, with the understanding that there are a few followup items listed in various comments

wrn14897 · 2025-11-04T21:11:57Z

LGTM, with the understanding that there are a few followup items listed in various comments

Thanks. I will file tickets for those followup items

…solve

vercel bot deployed to Preview October 25, 2025 23:47 View deployment

vercel bot deployed to Preview October 26, 2025 00:12 View deployment

wrn14897 changed the title ~~feat: add support for alert auto-resolve + Incident.io integration~~ (WIP) feat: add support for alert auto-resolve + Incident.io integration Oct 26, 2025

teeohhem marked this pull request as draft October 27, 2025 15:11

wrn14897 changed the title ~~(WIP) feat: add support for alert auto-resolve + Incident.io integration~~ feat: add support for alert auto-resolve + Incident.io integration Oct 27, 2025

wrn14897 marked this pull request as ready for review October 27, 2025 17:31

wrn14897 force-pushed the warren/introduce-alert-state-var-and-auto-resolve branch from 3e00bb0 to fe1b1f6 Compare October 27, 2025 21:54

vercel bot deployed to Preview October 27, 2025 21:58 View deployment

vercel bot deployed to Preview October 27, 2025 23:50 View deployment

vercel bot deployed to Preview October 28, 2025 00:01 View deployment

vercel bot deployed to Preview October 28, 2025 00:30 View deployment

vercel bot deployed to Preview October 28, 2025 08:32 View deployment

wrn14897 commented Oct 28, 2025

View reviewed changes

wrn14897 force-pushed the warren/introduce-alert-state-var-and-auto-resolve branch from f3a599e to 2119ef2 Compare October 28, 2025 09:00

vercel bot deployed to Preview October 28, 2025 09:03 View deployment

vercel bot deployed to Preview October 28, 2025 09:31 View deployment

wrn14897 force-pushed the warren/introduce-alert-state-var-and-auto-resolve branch from 487e4a5 to 1a5b028 Compare October 28, 2025 09:38

vercel bot deployed to Preview October 28, 2025 09:40 View deployment

wrn14897 force-pushed the warren/introduce-alert-state-var-and-auto-resolve branch from 1a5b028 to feb7176 Compare October 28, 2025 09:44

vercel bot deployed to Preview October 28, 2025 09:47 View deployment

vercel bot deployed to Preview October 28, 2025 10:01 View deployment

vercel bot deployed to Preview October 28, 2025 10:13 View deployment

hyperdxio deleted a comment from claude bot Oct 28, 2025

pulpdrew reviewed Oct 28, 2025

View reviewed changes

vercel bot deployed to Preview October 28, 2025 17:11 View deployment

vercel bot deployed to Preview October 28, 2025 17:32 View deployment

wrn14897 added 16 commits November 4, 2025 01:00

fix: eventId should be unique regardless of time window

69289c1

feat: implement auto resolve

908aa37

style: use if-else in the template for incident.io status mapping

80a72ce

docs: add changesets

e0a1f91

feedback: enrich eventId (dedupe key) + fetch state from alert history

91512ad

docs: add a comment

6e346c4

fix: handle group-by alerts (should fire/resolve multiple ones)

26f3028

fix: fill out missing group-by alert histories

1e5731c

ci: add more tests

2ebc903

feedback: address inconsistent histories vs alert state

2f8d73e

docs: update changeset

011759d

feedback: styles + types

bfc85b7

feedback: deprecate 'previous' and move codes to updateAlertState

e14aa18

feadback: handle hybrid group-by alertid in previousMap

5d77a0f

docs: add a comment

f9a2baa

ci: add a int test for multi buckets check

23a7e4b

wrn14897 force-pushed the warren/introduce-alert-state-var-and-auto-resolve branch from 025dd4e to 2f828ae Compare November 4, 2025 09:01

vercel bot had a problem deploying to Preview November 4, 2025 09:03 Failure

wrn14897 force-pushed the warren/introduce-alert-state-var-and-auto-resolve branch from 2f828ae to d3e29ad Compare November 4, 2025 09:09

vercel bot deployed to Preview November 4, 2025 09:13 View deployment

feedback: take the latest datetime from the group

2a3adce

wrn14897 force-pushed the warren/introduce-alert-state-var-and-auto-resolve branch from d3e29ad to 2a3adce Compare November 4, 2025 17:55

vercel bot deployed to Preview November 4, 2025 17:58 View deployment

pulpdrew approved these changes Nov 4, 2025

View reviewed changes

wrn14897 added the automerge label Nov 4, 2025

Merge branch 'main' into warren/introduce-alert-state-var-and-auto-re…

3f3231c

…solve

kodiakhq bot merged commit f612bf3 into main Nov 4, 2025
8 of 9 checks passed

kodiakhq bot deleted the warren/introduce-alert-state-var-and-auto-resolve branch November 4, 2025 22:53

vercel bot deployed to Preview November 4, 2025 22:53 View deployment

feat: add support for alert auto-resolve + Incident.io integration #1298

feat: add support for alert auto-resolve + Incident.io integration #1298

Uh oh!

Conversation

wrn14897 commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vercel bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review

Critical Issues

Important Issues

Minor Observations

Recommendation

Uh oh!

github-actions bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Results

Uh oh!

wrn14897 Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

pulpdrew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pulpdrew left a comment

Choose a reason for hiding this comment

Uh oh!

wrn14897 commented Nov 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wrn14897 commented Oct 25, 2025 •

edited

Loading

changeset-bot bot commented Oct 25, 2025 •

edited

Loading

vercel bot commented Oct 25, 2025 •

edited

Loading

claude bot commented Oct 25, 2025 •

edited

Loading

github-actions bot commented Oct 25, 2025 •

edited

Loading